Immigration flows in Greece, 2020-2021

The impact of COVID-19 on local agricultural economy

A Data Analytics project in Python, R, SQL and Microsoft Excel. Visualized and reported in R Markdown, ggplot2 and Tableau. Some CSS touches here and there. Educational and showcase purposes only.¹

last updated: 30 Sept. 2021 - changelog

Dependencies

If you plan to knit or reproduce by chunk the code in this file, you must download the raw *.rmd file from the dropdown Code menu at the top-right on this page. Also, this document comes with a list of required packages. Un-comment the ‘libraries chunk’ section in the raw rmd file to download the packages. If you plan though to read this report from this html document, no action is required.

1 Introduction

Fruits and vegetables output in Greece comprise almost half (43,5%) of the total agricultural output of the country². Nevertheless, like in all Mediterranean countries, crops show significant seasonality and are highly dependent on migrant workers. In southern EU countries, like Greece in particular, irregular and/or regular migrant workers meet the need for seasonal and permanent labor. In many cases, informal brokers play a major role in the recruitment of migrant laborers, most of whom work irregularly. ³

Greece’s ‘Requests for Hiring’ (RfH) legal framework (art. 11-14, law no.4251/2014 and art. 16, law no. 4783/2021) for the seasonal or full-time employment of third-country nationals has been proved in many occasions to underperform primarily by failing to secure employers’ needs, especially in the agricultural sector. Secondly, it renders the whole process quite unattractive for stakeholders involved (employers/foreign employees) to engage in legal interactions with the public agencies (Tax, Social Security, Law enforcement) due to high costs, bureaucracy and a long-established sentiment of depreciation and disbelief to the Greek Public Administration efficacy in solving problems.

During pro-Covid19 era, cultivating and harvesting needs were in general covered by legal staying but illegaly working Albanian workers who were freely crossing Greece’s national borders using the Schengen Treaty provision of ‘free 3 months stay, every 6 months, in European soil’. The problem is that this type of residence is not considered a residence permit with working rights because it explicitly prohibits any sort of employment. Nevertheless, this was for years the only way, although unofficial, for serving the needs of Greece’s agricultural economy. Under this scheme, both employers and workers were engaging in a labor system where all payments were unofficial and ‘under the table’, no social security or tax costs were paid and the Greek State never knew the actual number of people living and residing in its territory, working illegally.

2 The problem

Within this informal labor ‘framework’, Covid-19 created unprecedented working disturbances to the local agricultural economy which got disrupted by severe lack of working hands due to the closed borders between Greece and neighboring Albania. To remedy this, the Greek state issued law no.4783/2021(article 16) that allowed on an exceptional basis the invitation (‘requests for hiring’(RfH)) of unskilled Albanian land workers, using fast-track procedures to catch up with the pressing needs of the farming production in the field. So those workers that the previous years were working through ‘free circulation’ and under the ‘3 free months stay’, now they become ‘visible’ and obliged to pay fees and appear officially as documented unskilled workers. Similarly, employers declare this incoming workforce to authorities and thus break a rather ‘vicious circle’ of irregular working, payment and social security conditions.

This study examines comparatively the immigration flows in Greece for both pandemic years, 2020 and 2021 (end of August). The purpose of this study is to provide insights into the aforementioned problem by analyzing the current status of RfH in Greece, especially with regard to the COVID-19 pandemic situation. The goal is to provide the conceptual basis for a new legal framework on RfH that deters tax evasion and enhances the implementation of proper social security provisions, while at the same time facilitating the workload of law enforcement agencies in securing borders’ safety and domestic order. By analyzing available data, there will be an effort to standardize common trends within immigration flows and better organize the invitation system for the years to come. Also, an attempt will be made for a rough calculation of profits, in terms of actual economic gains, for the Greek state.

2.1 Data Discovery

Data for this project was collected by the Ministry of Migration & Asylum official website and the Open Data repository of the Greek Government. For more details about steps involved for this phase, you can click the images or download the full technical bulletins⁴: a) the Data Discovery phase report and b) the corresponding Scope of Work (Sow).

Figure 2.1: Data discovery report

Figure 2.2: Scope of Work(SoW)

2.2 Data Preparation

2.2.1 Files structure

The client provided us with a total of 240 files (pdf and xlsx) organized into 78 date folders, containing data regarding RfH⁵ in Greece, for 2020 (160 files and 53 folders) and for 2021 (up to the end of August, 80 files and 23 folders).

Table 2.1: Total files (dirty)
Year	Files	Folders
2020	160	80
2021	53	23

2.2.2 File handling

Several file handling interventions were deemed necessary due to the extended and complex directory structure in which raw data were given to us. We used Python to clean things up.

First, we had to identify and separate the *.xlsx from *.pdf files and drop the latter. For the year 2020, 107 *.pdf files were identified and 46 *.pdf files for 2021.

> import os, fnmatch
> 
> os.chdir('/datasets')
> 
> count = 0
> for dirpath, dirnames, filenames in os.walk(os.curdir):    
>     for file in filenames:        
>         if fnmatch.fnmatch(file, '*.pdf'):
>             print(file)
>             count += 1
> print('Total pdf files found:', count)

Second, from the initial total of 240 files, 153 were eventually deleted, being left with 53 datasets for 2020 and 34 datasets for 2021.

> import os, fnmatch
> 
> os.chdir('/datasets')
> 
> count = 0
> for dirpath, dirnames, filenames in os.walk(os.curdir):    
>     for file in filenames:        
>         if fnmatch.fnmatch(file, '*.pdf'):
>             print(file)
>             count += 1
>             os.remove(os.path.join(dirpath, file))

So 87 datasets were left scattered in 78 folders. Also, file naming issues came up since no consistent file naming convention was used during the creation of the Excel datasets.

Figure 2.3: The file naming issue: dozens of datasets without consistent naming

Third, we simplified the file structure by moving all scattered Excel datasets from the 78 folders into one folder, named data.

> import shutil, os
> 
> src = '/datasets'
> dest = '/data'
> 
> count = 0
> for root, subdirs, files in os.walk(src, topdown=True):
>     for file in files:
>         source = os.path.join(root, file)
>         count += 1
>         shutil.move(source, dest)

Fourth, we dropped a series of files containing duplicate information for the same day of recorded data, using part of their filename as a conditional criterion in Python (AADE).

> import os
> 
> os.chdir('/Users/George/Documents/bio/portfolio/Case Studies/01 - Capstone/data/')
> count = 0
> for f in os.listdir():
>     file_name = os.path.splitext(f)[0]
>     count += 1
>     aad = file_name.split()
>     if len(aad) > 1:
>         if(aad)[1] == 'ΑΑΔΕ':
>             os.remove(f)

Fifth, in order to maintain initial (raw) directory structure semantics (i.e., year_folder –> date_folders –> dataset files), we dropped all initial file names and replaced them by constructs based on their date-modified property from the os.stat module in Python, after adding a preceding and gradually incremented number.

> 
> import os
> from datetime import date
> 
> os.chdir('/data')
> 
> files = os.listdir()
> 
> count = 0
> for f in files:
>     file_name = os.path.splitext(f)[0]
>     file_ext = os.path.splitext(f)[1]
>     mod_time = os.stat(f).st_mtime
>     file_date = date.fromtimestamp(mod_time)
>     count += 1
>     new_name = ('{}_{}{}').format(count, file_date, file_ext)
>     os.rename(f, new_name)

This is the whole transformation process, from the initial reception of the raw data (left image), to their copying into one folder and deletion of pdf files (middle image), up to their consistent renaming (right image):

Figure 2.4: File structure of raw data

Figure 2.5: File structure transformation

Figure 2.6: File naming transformation

Sixth, we combined all 87 datasets into one file (new_dataset.xlsx below) and then checked their variables for any commonalities or discrepancies.

> mydir = "data"
> myfiles = list.files(path=mydir, pattern="*xlsx", full.names=TRUE)
> dat_xlsx = ldply(myfiles, read_xlsx)
> colnames(dat_xlsx)
> new_dataset <- writexl::write_xlsx(dat_xlsx, "new_dataset.xlsx")
>

Finally, I imported the final working *.xlsx dataset into R for data cleaning checks and renamed its variables to a more human-friendly naming convention (removed the underscores).

> mydf <- readxl::read_xlsx("rfh.xlsx")
> colnames(mydf)
> rfh_renamed <- rename(mydf, "nl_no" = "nlno", "nl_yes" = "nlyes", "nl_total" = "nltotal", 
+                       "municipality" = "muni", "address" = "addr")
> writexl::write_xlsx(rfh_renamed, "rfh4.xlsx")
> colnames(rfh_renamed)

For more details about data preparation, please click on the thumbnail below or download the full technical report.

Figure 2.7: Data cleaning report

2.3 Data Cleaning

Data received was more or less clean due to the nature of their origin (a well-maintained database where values entered adhere to formatting and conceptual constraints). Nevertheless, a few interventions were necessary like the deletion of 1101 blank cells (Microsoft Excel (=COUNTBLANK)) and the transliteration of some variables (renaming of columns) to latin characters.

For more details on our data cleaning approach, please click on the thumbnails below or download the full technical report.

Figure 2.8: Data cleaning report (1)

Figure 2.9: Data cleaning report (2)

2.4 Data Analysis & Visualization

According to the analysis, for the “COVID19” years 2020 and 2021 (up to the end of August), 25000 employers requested the hiring of 61784 third-country nationals, mostly Albanians and more or less evenly shared between those 2 years.

2.4.1 Rfh by Category (2020 and 2021 (end of August))

So during the time period where free circulation through Greek-Albanian borders (where the main immigration flows are originating from) was forbidden, labor force was only allowed to move from Albania to Greece under the general ‘Request for Hiring’ (art. 11-15, law no.4251/2014) procedure, which consisted of the following categories and 61784 workers:

> SELECT SUM (nlyes) AS total, category 
+ FROM rfh
+ GROUP BY category ORDER BY SUM(nlyes) DESC

Data confirmed the empirical knowledge that indeed, the newly formed covid19-proc legal framework dominated the RfHs fulfilled between Greece and Albania and supported in effect the needs of agricultural economy in Greece those years.

2.4.2 Distribution of COVID19 special procedure workers by geographical region

Almost 50000 Albanians worked in Greece the first two COVID19 years with the covid19-proc procedure, with the majority of those working in the region of Central Macedonia (Edessa, Polygyros, Veroia).

> SELECT SUM(nlyes) AS total, agency
+ FROM rfh
+ WHERE category = 'covid19-proc'
+ GROUP BY agency
+ ORDER BY total DESC

The following visualization from Tableau can clearly indicate the distribution of workforce around Greece’s territory, for the years under research. Please click the thumbnail to enlarge or visit the interactive dashboard at Tableau platform.

Figure 2.10: Distribution of third-country nationals workforce at Greece, during 2020 and 2021

2.4.3 RfH (COVID19 special procedure - employees)

Six prefectures of the Region of Central Macedonia (Edessa, Polygyros, Veroia, Katerini, Serres, Kilkis) comprise the 68.22% of total RfHs.

> rfh %>% 
+   filter(agency == c('ΕΔΕΣΣΑ', 'ΠΟΛΥΓΥΡΟΣ', 'ΒΕΡΟΙΑ', 'ΚΑΤΕΡΙΝΗ', 'ΣΕΡΡΕΣ', 'ΚΙΛΚΙΣ')) %>% 
+   ggplot()+
+   geom_bar(aes(x = agency))

2.4.4 Estimated profits for the Greek economy

For those 50372 third-country workers (mostly Albanians), their respective employers were obliged for the first time of this kind of seasonal framework to pay an entrance fee of 100 euros. Also, the social security status of these workers had to be taken care of, meaning about 135 euros per person monthly. Simply put, this exceptional procedure of RfH, not only protected and served the urgent needs of agricultural economy of Greece, mainly of its northern regions as it was shown, but it also secured more than 11.000.000 euros for the Greek state for both years combined.

For more details on our data analysis and data visualization approach, please click on the thumbnails below or download the data analysis and data visualization report.

Figure 2.11: Data analysis report

Figure 2.12: Data Visualization report

3 Conclusion

Data analysis has clearly concluded to the following:

Greece is lacking about 25000 workers yearly from its own domestic agricultural force. Crops yield thus can be only secured via the help of foreign, mostly Albanian workers.
COVID19 years proved that a well-coordinated state effort can aggregate profits that can surpass 5.000.000 euros per year where the pre-COVID19 years these profits were completely lost (zero).
For the first time, a workflow of more than 50000 third-country nationals was properly recorded and documented, thus leading to unprecedented administrative and economical gains for the Greek state.

4 Disclaimer

No official agencies or stakeholders were involved for the preparation, implementation, execution and completion of this project. It serves as an educational showcase only on Data Analytics technologies. Data do not correspond to real events, persons or public bodies.

5 Session Info

> devtools::session_info()
## - Session info ---------------------------------------------------------------
##  setting  value                       
##  version  R version 4.1.0 (2021-05-18)
##  os       Windows 10 x64              
##  system   x86_64, mingw32             
##  ui       RTerm                       
##  language (EN)                        
##  collate  Greek_Greece.1253           
##  ctype    Greek_Greece.1253           
##  tz       Europe/Istanbul             
##  date     2021-09-30                  
## 
## - Packages -------------------------------------------------------------------
##  package      * version date       lib source        
##  assertthat     0.2.1   2019-03-21 [1] CRAN (R 4.1.0)
##  backports      1.2.1   2020-12-09 [1] CRAN (R 4.1.0)
##  base64enc      0.1-3   2015-07-28 [1] CRAN (R 4.1.0)
##  bit            4.0.4   2020-08-04 [1] CRAN (R 4.1.0)
##  bit64          4.0.5   2020-08-30 [1] CRAN (R 4.1.0)
##  blob           1.2.2   2021-07-23 [1] CRAN (R 4.1.0)
##  bookdown       0.23    2021-08-13 [1] CRAN (R 4.1.1)
##  broom          0.7.8   2021-06-24 [1] CRAN (R 4.1.0)
##  bslib          0.3.0   2021-09-02 [1] CRAN (R 4.1.1)
##  bsplus         0.1.2   2020-06-25 [1] CRAN (R 4.1.1)
##  cachem         1.0.5   2021-05-15 [1] CRAN (R 4.1.0)
##  callr          3.7.0   2021-04-20 [1] CRAN (R 4.1.0)
##  cellranger     1.1.0   2016-07-27 [1] CRAN (R 4.1.0)
##  cli            3.0.1   2021-07-17 [1] CRAN (R 4.1.0)
##  colorspace     2.0-2   2021-06-24 [1] CRAN (R 4.1.0)
##  crayon         1.4.1   2021-02-08 [1] CRAN (R 4.1.0)
##  DBI            1.1.1   2021-01-15 [1] CRAN (R 4.1.1)
##  dbplyr         2.1.1   2021-04-06 [1] CRAN (R 4.1.0)
##  desc           1.3.0   2021-03-05 [1] CRAN (R 4.1.0)
##  devtools       2.4.2   2021-06-07 [1] CRAN (R 4.1.0)
##  digest         0.6.27  2020-10-24 [1] CRAN (R 4.1.0)
##  downloadthis * 0.2.1   2020-09-17 [1] CRAN (R 4.1.1)
##  dplyr        * 1.0.7   2021-06-18 [1] CRAN (R 4.1.0)
##  ellipsis       0.3.2   2021-04-29 [1] CRAN (R 4.1.0)
##  evaluate       0.14    2019-05-28 [1] CRAN (R 4.1.0)
##  fansi          0.5.0   2021-05-25 [1] CRAN (R 4.1.0)
##  farver         2.1.0   2021-02-28 [1] CRAN (R 4.1.0)
##  fastmap        1.1.0   2021-01-25 [1] CRAN (R 4.1.0)
##  forcats      * 0.5.1   2021-01-27 [1] CRAN (R 4.1.0)
##  fs             1.5.0   2020-07-31 [1] CRAN (R 4.1.0)
##  generics       0.1.0   2020-10-31 [1] CRAN (R 4.1.0)
##  ggplot2      * 3.3.5   2021-06-25 [1] CRAN (R 4.1.0)
##  glue           1.4.2   2020-08-27 [1] CRAN (R 4.1.0)
##  gtable         0.3.0   2019-03-25 [1] CRAN (R 4.1.0)
##  haven          2.4.1   2021-04-23 [1] CRAN (R 4.1.0)
##  here         * 1.0.1   2020-12-13 [1] CRAN (R 4.1.1)
##  highr          0.9     2021-04-16 [1] CRAN (R 4.1.0)
##  hms            1.1.0   2021-05-17 [1] CRAN (R 4.1.0)
##  htmltools    * 0.5.2   2021-08-25 [1] CRAN (R 4.1.1)
##  httr           1.4.2   2020-07-20 [1] CRAN (R 4.1.0)
##  jquerylib      0.1.4   2021-04-26 [1] CRAN (R 4.1.0)
##  jsonlite       1.7.2   2020-12-09 [1] CRAN (R 4.1.0)
##  kableExtra   * 1.3.4   2021-02-20 [1] CRAN (R 4.1.1)
##  knitr        * 1.33    2021-04-24 [1] CRAN (R 4.1.1)
##  labeling       0.4.2   2020-10-20 [1] CRAN (R 4.1.0)
##  lattice        0.20-44 2021-05-02 [2] CRAN (R 4.1.0)
##  lifecycle      1.0.0   2021-02-15 [1] CRAN (R 4.1.0)
##  lubridate      1.7.10  2021-02-26 [1] CRAN (R 4.1.0)
##  magrittr       2.0.1   2020-11-17 [1] CRAN (R 4.1.0)
##  Matrix         1.3-3   2021-05-04 [2] CRAN (R 4.1.0)
##  memoise        2.0.0   2021-01-26 [1] CRAN (R 4.1.0)
##  mime         * 0.11    2021-06-23 [1] CRAN (R 4.1.0)
##  modelr         0.1.8   2020-05-19 [1] CRAN (R 4.1.0)
##  munsell        0.5.0   2018-06-12 [1] CRAN (R 4.1.0)
##  pillar         1.6.2   2021-07-29 [1] CRAN (R 4.1.0)
##  pkgbuild       1.2.0   2020-12-15 [1] CRAN (R 4.1.0)
##  pkgconfig      2.0.3   2019-09-22 [1] CRAN (R 4.1.0)
##  pkgload        1.2.1   2021-04-06 [1] CRAN (R 4.1.0)
##  png            0.1-7   2013-12-03 [1] CRAN (R 4.1.0)
##  prettyunits    1.1.1   2020-01-24 [1] CRAN (R 4.1.0)
##  processx       3.5.2   2021-04-30 [1] CRAN (R 4.1.0)
##  ps             1.6.0   2021-02-28 [1] CRAN (R 4.1.0)
##  purrr        * 0.3.4   2020-04-17 [1] CRAN (R 4.1.0)
##  R6             2.5.1   2021-08-19 [1] CRAN (R 4.1.0)
##  Rcpp           1.0.7   2021-07-07 [1] CRAN (R 4.1.0)
##  readr        * 2.0.1   2021-08-10 [1] CRAN (R 4.1.1)
##  readxl       * 1.3.1   2019-03-13 [1] CRAN (R 4.1.1)
##  remotes        2.4.0   2021-06-02 [1] CRAN (R 4.1.0)
##  reprex         2.0.0   2021-04-02 [1] CRAN (R 4.1.0)
##  reticulate     1.20    2021-05-03 [1] CRAN (R 4.1.1)
##  rlang          0.4.11  2021-04-30 [1] CRAN (R 4.1.0)
##  rmarkdown      2.10    2021-08-06 [1] CRAN (R 4.1.0)
##  rmdformats     1.0.2   2021-04-19 [1] CRAN (R 4.1.1)
##  rprojroot      2.0.2   2020-11-15 [1] CRAN (R 4.1.0)
##  RSQLite      * 2.2.8   2021-08-21 [1] CRAN (R 4.1.1)
##  rstudioapi     0.13    2020-11-12 [1] CRAN (R 4.1.0)
##  rvest          1.0.0   2021-03-09 [1] CRAN (R 4.1.0)
##  sass           0.4.0   2021-05-12 [1] CRAN (R 4.1.1)
##  scales         1.1.1   2020-05-11 [1] CRAN (R 4.1.0)
##  sessioninfo    1.1.1   2018-11-05 [1] CRAN (R 4.1.0)
##  stringi        1.6.2   2021-05-17 [1] CRAN (R 4.1.0)
##  stringr      * 1.4.0   2019-02-10 [1] CRAN (R 4.1.0)
##  svglite        2.0.0   2021-02-20 [1] CRAN (R 4.1.1)
##  systemfonts    1.0.2   2021-05-11 [1] CRAN (R 4.1.1)
##  testthat       3.0.4   2021-07-01 [1] CRAN (R 4.1.0)
##  tibble       * 3.1.3   2021-07-23 [1] CRAN (R 4.1.0)
##  tidyr        * 1.1.3   2021-03-03 [1] CRAN (R 4.1.0)
##  tidyselect     1.1.1   2021-04-30 [1] CRAN (R 4.1.0)
##  tidyverse    * 1.3.1   2021-04-15 [1] CRAN (R 4.1.1)
##  tzdb           0.1.2   2021-07-20 [1] CRAN (R 4.1.0)
##  usethis        2.0.1   2021-02-10 [1] CRAN (R 4.1.0)
##  utf8           1.2.2   2021-07-24 [1] CRAN (R 4.1.0)
##  vctrs          0.3.8   2021-04-29 [1] CRAN (R 4.1.0)
##  viridisLite    0.4.0   2021-04-13 [1] CRAN (R 4.1.0)
##  webshot        0.5.2   2019-11-22 [1] CRAN (R 4.1.1)
##  withr          2.4.2   2021-04-18 [1] CRAN (R 4.1.0)
##  writexl        1.4.0   2021-04-20 [1] CRAN (R 4.1.1)
##  xfun         * 0.25    2021-08-06 [1] CRAN (R 4.1.1)
##  xml2           1.3.2   2020-04-23 [1] CRAN (R 4.1.0)
##  yaml           2.2.1   2020-02-01 [1] CRAN (R 4.1.0)
## 
## [1] C:/Users/George/Documents/R/win-library/4.1
## [2] C:/Program Files/R/R-4.1.0/library

6 Changelog

300921 website published.

This is an R Markdown document. To lean more about ‘R Markdown’ formatting, read here ↩︎
https://ec.europa.eu/info/sites/default/files/food-farming-fisheries/farming/documents/agri-statistical-factsheet-el_en.pdf ↩︎
Papadopoulos Apostolos, Fratsea Loukia-Maria. (2016). “Migrant labour and intensive agricultural production in Greece: the case of Manolada strawberry industry, ResearchGate.↩︎
The Data Discovery report is part of my ‘(not) A Handbook for Data Analytics’ project. Visit project’s site for more details.↩︎
Requests for Hiring (“μετακλήσεις” in Greek, art. 11-15, law no.4251/2014)↩︎